Dynamic customizable human-computer interaction behavior

ABSTRACT

Systems and methods for customizing behavior of a computing system based on details of interactions with the computing system by a user, such as a direction, intensity, or magnitude of a particular input from a user input device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. §119(e)of U.S. Provisional Application No. 61/425,156, filed Dec. 20, 2010, thedisclosure of which is hereby incorporated by reference in its entirety.

BACKGROUND

There are many situations in which users employ computing systems toview information where it is important that the users accurately andefficiently interpret that information. Current computing systems arecapable of presenting information at rates that exceed a user'sperceptual ability to accurately interpret the information presented,resulting in errors. This problem will increase in severity as computerprocessing power and bandwidth continue to increase.

SUMMARY

For purposes of this summary, certain aspects, advantages, and novelfeatures are described herein. It is to be understood that notnecessarily all such advantages may be achieved in accordance with anyparticular embodiment. Thus, for example, those skilled in the art willrecognize that the invention may be embodied or carried out in a mannerthat achieves one advantage or group of advantages as taught hereinwithout necessarily achieving other advantages as may be taught orsuggested herein.

Disclosed herein are systems and methods for operating a computingdevice so that the speed of information presentation is matched to theuser's preferences and/or to various factors that may impact the user'sability to accurately and efficiently interpret it, including propertiesof the visual information, properties of the display device,environmental factors, characteristics of the user, such as expertiseand fatigue, and any other factors that may be relevant to the user'sreview of the information.

In one embodiment, a computing system comprises one or more hardwareprocessors configured to execute software instructions stored in modulesand a tangible computer readable medium storing modules configured forexecution by the one or more hardware processors. In one embodiment, themodules include a display module configured to display medical images ona display device of the computing system and a control module configuredto: access a data structure storing conditions associated withrespective behavior models; identify one or more of the conditions thatare matched by one or more characteristics associated with exam viewing,the computing system, an environment surrounding the computing system,the display device, bandwidth available to the computing system, themedical images, a patient, a medical history of the patient, an inputdevice, and/or a user of the computing system, wherein at least one ofthe identified conditions is associated with a display behaviorincluding two or more rates of display of medical images associated withrespective levels of interaction with an input device; receive data froman input devices indicating a level of interaction with the inputdevice; determine a rate of display of medical images based on anassociation in the at least one of the identified conditions between thelevel of interaction and one of the two or more rates of display ofmedical images; and display the medical images at the determined rate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating one embodiment of a computingsystem that may be used to implement certain systems and methodsdescribed herein.

FIG. 2 illustrates an input device in which the user tilts a knob andthe degree of the tilt is provided as input to a computing device.

FIG. 3 is a graphical representation of information within aninteraction behavior model, graphing the relationship between imagenavigation speed in images/second vs. the angle of tilt of an inputdevice, such as the one shown in FIG. 2.

FIG. 4 shows examples of information within one or more interactionbehavior models, mapping user input from an input device to the speed ofimage navigation on a computing device.

FIG. 5 a illustrates information within an interaction behavior model.

FIG. 5 b illustrates information within an interaction behavior model.

FIG. 5 c illustrates an interaction behavior model that maps input to anoutput that can be interpreted by a computing device.

FIG. 6 is a block diagram depicting one embodiment of a system using theinteraction behavior model.

FIG. 7 illustrates a computing device with a display screen and a sensorthat detects the tilt of the device and depicts how the device can betilted by the user to provide input using an embodiment of theinteraction behavior model.

FIG. 8 shows different modes of user input, tilting, rotation, andtranslation.

FIG. 9 illustrates a mouse that includes two buttons and wheel.

FIG. 10 illustrates a handheld computing device such as a smartphone,PDA or tablet computer that includes a touch screen.

FIG. 11 illustrates a user interacting with a tablet computer or displaywith a touch screen.

FIG. 12 illustrates one embodiment of an interaction behavior model thatmay be used to implement certain systems and methods described herein.

FIG. 13 displays six image frames as might be displayed on a displaydevice, where each image frame displays an image from a different seriesfrom within one or more medical imaging exams.

FIG. 14 shows aspects of an embodiment of an interaction behavior modelused with digital pathology.

FIG. 15 a shows aspects of embodiments of an interaction behavior modelin which a region of an image has been marked by CAD.

FIG. 15 b shows aspects of embodiments of an interaction behavior modelin which a region of an imaging volume have been marked by CAD.

FIG. 16 shows aspects of an embodiment of an interaction behavior modelin which a region of a mammo tomosynthesis exam has been marked by CAD.

These and other features will now be described with reference to thedrawings summarized above. The drawings and the associated descriptionsare provided to illustrate certain embodiments of the invention and notto limit the scope of the invention. Throughout the drawings, referencenumbers may be re-used to indicate correspondence between referencedelements.

DETAILED DESCRIPTION

Embodiments of the disclosure will now be described with reference tothe accompanying figures, wherein like numerals refer to like elementsthroughout. The terminology used in the description presented herein isnot intended to be interpreted in any limited or restrictive manner,simply because it is being utilized in conjunction with a detaileddescription of certain specific embodiments of the disclosure.Furthermore, embodiments of the disclosure may include several novelfeatures, no single one of which is solely responsible for its desirableattributes or which is essential to practicing the embodiments of thedisclosure herein described.

As used herein, the terms “viewer” and “user” are used interchangeablyto describe an individual (or group of individuals) that interfaces witha computing device. Users may include, for example, doctors,radiologists, hospital staff, or other individuals involved inacquisition, analysis, storage, management, or other tasks related tomedical images. Any discussion herein of user preferences should beconstrued to also, or alternatively, include user group preferences,site preferences, system preferences, and/or default softwarepreferences.

Depending on the embodiment, the methods described with reference to theflowcharts, as well as any other methods discussed herein, may includefewer or additional blocks and/or the blocks may be performed in adifferent order than is illustrated. Software code configured forexecution on a computing device in order to perform the methods may beprovided on a computer readable medium, such as a compact disc, digitalvideo disc, flash drive, hard drive, memory device or any other tangiblemedium. Such software code may be stored, partially or fully, on amemory of a computing device (e.g., RAM, ROM, etc.), such as thecomputing system 150 (see discussion of FIG. 1, below), and/or othercomputing devices illustrated in the figures, in order to perform therespective methods. For ease of explanation, the methods will bedescribed herein as performed by the computing system 150, but themethods are not limited to performance by the computing system 150 andshould be interpreted to include performance by any one or more of thecomputing devices noted herein and/or any other suitable computingdevice.

Images

In the fields of radiology, cardiology, and pathology, for example,physicians often view a large amount of imaging information and it iscritical that they accurately interpret the imaging information to makean accurate diagnosis. In addition, there are many other fields whereaccurate and efficient interpretation of imaging information isimportant, such as baggage screening, satellite imaging, seismic imagingused in oil and gas exploration, and surveillance video.

Medical imaging exams can be acquired by a number of different medicalimaging techniques, including computed tomography (CT), magneticresonance imaging (MRI), ultrasound, nuclear medicine, positron emissioncomputed tomography (PET), digital angiography, mammography, computedradiography, digital radiography, fluoroscopy, and others such as imagesgenerated in medical pathology and endoscopy. A variety of computingsystems are used to manage medical images, including storage,distribution, analysis, processing and display. These computing systemsinclude Picture Archive and Communication Systems (PACS), DigitalPathology Systems, Cardiovascular Information Systems, Computer AidedDiagnosis Systems (CAD), 3D Processing systems, Electronic MedicalRecord (EMR) systems, standalone software for display of medical images,web based Personal Health Record (PHR) systems and other systems thatmanage medical imaging exams, such as online physician portals.

As described below, physicians and others utilize computing devices,herein referred to as information display computing devices, to viewinformation. Information display computing devices can come in manyforms and can be single computing devices or combinations of computingdevices, including dedicated PACS workstations, Electronic MedicalRecord Systems, general purpose computing systems, computer tablets,and/or handheld devices such as smartphones.

Medical imaging exams often include a large number of images. Forexample, a computed tomography (CT) exam may include hundreds orthousands of images. Because it is usually impractical to view all of anexam's images simultaneously, the display of an information displaycomputing device typically displays a fraction of the total number ofimages at once and allows the user to interact with the informationdisplay computing device to display other images or other portions of animage being displayed.

In addition, a medical image may include too many pixels to be displayedat the full or desired resolution on an information display computingdevice. For example, a single digital pathology image could include atrillion pixels, vastly exceeding the display resolution of a 1megapixel monitor. The mismatch between the pixel size of the image andthe pixel size of the display device requires the user to interact withthe computing device to display various portions of the image.

Medical imaging exams are often divided into series, where a seriescomprises a group of one or more images acquired or displayed using acertain technique. Images within a series may differ in anatomicposition or time of acquisition, for example. For example, in a computedtomography exam, one series might include 250 contiguous 0.6 mm thickimages of the brain obtained without administration of intravenouscontrast material. Another series might include 250 contiguous 0.6 mmthick images of the brain obtained during administration of intravenouscontrast material. A brain MRI might include multiple series acquiredwith different technical parameters, possibly including images acquiredbefore administration of intravenous contrast material and imagesacquired after administration of intravenous contrast material. Forexample a brain MRI might include the following series: Sagittal T1,axial T1, axial FLAIR, axial T2, axial diffusion, coronal gradient echoand post-contrast axial, sagittal and coronal T1 weighted images.

Separate image series may be displayed in separate display frames on adisplay device, e.g., as illustrated in FIG. 13 where six series from abrain MRI are displayed as they might appear on an information displaycomputing device. Depending on the input device and user preference, anumber of methods can be used to allow the user to change the imagewithin the series that is displayed in an image frame. The images withina series are typically numbered, for example 1 to 100 in a series with100 images.

While much of the discussion herein refers to display of medical images,the systems and methods disclosed herein are not limited to such images.In fact, the systems and methods discussed herein may be applied to anytype of information that is presented visually and controlled by anyinput device. For example, the systems and methods discussed herein maybe used with images related to baggage screening, satellite imaging,seismic imaging used in oil and gas exploration, surveillance video,and/or any other type of images. Additionally, while visual informationmay be in the form of images, visual information may be in other forms.For example, visual information may be in graphical form, such as EKGand EEG information. Visual information may also be in the form of text,such as medical laboratory results, medical exam reports or documentsutilized in legal proceedings. Visual information may also be presentedin other forms or as a mixture of various forms, for example amultimedia web page comprising text, graphics, images, and/oranimations. The systems and methods described here may be applied to anyvisual information presented on a computing device. Thus, any referenceherein to medical images should be construed to cover other embodimentsinvolving other image types.

Example Features of Certain Embodiments

In some embodiments, users may interact with computing systems via awide range of input devices to control the presentation of informationand/or to perform other functions. For any manipulation of a particularinput device by a user, there are many potential actions that acomputing device might take. For example, an input device might beutilized by a user to control the display of visual information on acomputing device. In some embodiments, the systems and methods discussedherein modify the behavior of the computing device as a function of userinput, accounting for factors such as properties of the visualinformation displayed, properties of the display device, environmentalfactors, user preference, and/or characteristics of the user such asexpertise and fatigue.

In addition, users may interact with computing systems via a wide rangeof input devices to control the presentation of information or toperform other functions using computing devices. For any manipulation ofa particular input device by a user, there are many potential actionsthat a computing device might take. For example, a user moving ajoystick type device 5 degrees to the right center could result in manydifferent actions. For example, it might result in serial presentationof images at a rate of 5 images/second, 10 images/second or 20images/second.

There is a need for better methods for matching a user's manipulation ofan input device to the resulting action that accounts for a variety offactors, including user preferences, properties of the device, and/orvarious factors related to the activity being controlled by the inputdevice. For example, in the case where the input device is used tocontrol a physical action, it may be useful for the mapping of userinput to an activity controlled by the computing device so that theactivity may be altered based on user preference, environmental factors,and/or characteristics of the user such as expertise and fatigue.

As used herein, the term “interaction behavior model” describes a model,algorithm, and/or other logic that may be used to customize display ofinformation on a computing device, customize the display of informationas a function of user input, and/or customize activity controlled by acomputing device as a function of user input. An interaction behaviormodel may be used to control the speed of presentation of informationpresented by a computing device independent of the input device. Inother embodiments, an interaction behavior model could be used tocontrol how input devices result in actions performed by computingdevices. In other embodiments, an interaction behavior model could beused to control how input from input devices is interpreted by computingdevices. Interaction behavior models may access any number of inputs inorder to determine how the computing device displays data and/orinteracts with a user.

Various adjustments may be made by a computing system applying aninteraction behavior model in response to user navigation input (e.g.,via an input device). For example, adjustments may include:

-   -   Which image or portion of an image is displayed.    -   Image display characteristics such as brightness/contrast,        window/level, magnification, panning.    -   Display parameters such as image view angle for computer        generated 3D volumetric images and other renderings.    -   Spatial position and direction in 3D volumetric endoluminal fly        through imaging as used in virtual colonography.    -   Parameters used in various types of image rendering, e.g.,        location and/or angle of a reconstruction plane in multiplanar        reconstruction (MPR) and maximum intensity projection (MPR)        reconstruction.    -   Spatial position and/or view angle in computer generated virtual        environments.

In addition, an interaction behavior model could be used to control howuser input via an input device is mapped into physical actionscontrolled by a computing device such as a machine or vehicle. Forexample, in various embodiments an interaction behavior model could beused to control how user input via a foot pedal or other input devicecontrols the speed or steering of a car, boat, aircraft, spacecraft,submarine, robot or drone. Interaction behavior models may also be usedin videogames and computer simulations, for example.

Example Factors

As noted above, various factors may be used by the computing device tocustomize operations performed by the computing device using aninteraction behavior model, such as how the computing device responds toinput received from one or more input devices from a particular user ina particular environment. For example, when a certain condition is met,e.g., one or more factors match a rule associated with a condition (seeFIG. 5 and beyond), the effect of particular user inputs (e.g.,movements of an input device) may be customized based on behaviorsand/or modifiers associated with the condition. Thus, conditions mayinclude any one or more thresholds of a factor (e.g., time of day isbefore 8 am or user has more than 10 years experience reading aparticular exam type) and/or an indication of whether or not a factor ispresent (e.g., the image has not been previously viewed). Use ofconditions that are based on various factors is discussed in furtherdetail below.

Below are several factors that may be used in customizing interactionbehavior models and/or selected parameters of interaction behaviormodels. Any one or more of these factors, and/or any other factors, maybe used in customizing actions of a computing device in accordance withthe systems and methods described herein.

-   -   Factors related to the user        -   Level of expertise, e.g., lower level of expertise may            require longer viewing times of medical imaging exams        -   Level of fatigue, e.g., number of hours worked that day,            time of day, measures of user fatigue.        -   User preference.    -   Factors related to exam viewing        -   Complete read vs. viewing with a specific goal, e.g., a            radiologist performing a diagnostic read on a brain MRI may            prefer to view it at a slower rate than a neurosurgeon that            has read the neuroradiologist's report and is primarily            interested in the size and location of an intracranial            hematoma for surgical decision making.        -   Viewing for complete read vs. for comparison purposes, e.g.,            a radiologist doing a primary interpretation of a chest CT            may prefer to view it at a slower rate than a prior            comparison exam that has already been interpreted.        -   Whether or not the current image has been viewed by the            user. For example, one embodiment of an interaction behavior            model might cap the image display rate at 5 images/second            for unviewed images, but allow an image display rate of 20            images/second for images that have been viewed by the user.    -   Environmental factors        -   Room noise that could distract the user.        -   Ambient light that could make viewing of images more            difficult.    -   Factors related to the display device        -   Spatial resolution.        -   Contrast resolution.        -   Background luminance and luminance range.        -   Brightness.    -   Characteristics of the Computing Device and Network Bandwidth.        -   Bandwidth, e.g., a user may prefer a constant lower rate of            image display than an irregular rate of display of            sequential images that could be caused by a low bandwidth            connection used to retrieve the images.        -   Computational speed.        -   Memory available for image preloading.    -   Factors related to the images being viewed        -   Image size in pixels (larger number of pixels may contain            more information)        -   Signal to Noise Ratio (SNR)        -   The type of imaging exam being viewed, e.g., MRI, CT,            nuclear medicine, etc.        -   Fraction of the image that is not homogeneous.    -   Factors related to the patient or other information related to        the images        -   Clinical indication for the exam, such as evaluation of            metastatic disease which may indicate that the probability            of abnormalities is higher than a routine screening exam.        -   Prior exams. For example, in one embodiment, the interaction            behavior model may be used to automatically slow the display            rate of images in anatomic regions where abnormalities were            identified on prior exams.        -   Computer Aided Diagnosis (CAD). For example, in one            embodiment, the interaction behavior model may utilize CAD            information or other forms of image analysis to slow display            in regions flagged as suspicious. For example, CAD data            indicating possible lesions in the liver may cause the            interaction behavior model to slow down viewing of images            including the liver, even if the user is providing the same            input for proceeding through the images. For example, CAD            data indicating possible lesions in virtual colonography may            cause the interaction behavior model to slow down the rate            of change of spatial position and direction in 3D volumetric            endoluminal fly through imaging as used in virtual            colonography.    -   Factors related to the way images are presented to the user        -   Image display size, e.g., larger size may require greater            search time due to limited perceptual field of view.        -   Image display parameters, e.g., a greater fraction of an            abdominal CT image may be relevant when displaying images            with soft tissue windows compared to bone windows.        -   Use of image coupling where more than one image is changing            on the screen, requiring the user to divide his attention            between images    -   Input device characteristics        -   Precision with which the user can manipulate the display            device, e.g., it can be difficult for users to precisely            manipulate a joystick to a particular position but            positioning it to within a range may be relatively easy

The mapping of physical actions to computer behavior could becustomized. For example, one user might prefer a forward tilt toincrement image number while another might prefer the opposite. One usermight want a forward tilt to navigate superiorly within the bodyregardless of how the patient was scanned.

Example Computing System

FIG. 1 is a system diagram which shows the various components of asystem 100 for displaying information utilizing certain systems andmethods described herein. As shown, the system 100 may include aninformation display computing device 150 (also referred to herein as a“computing device 150”) and may include other systems, including thoseshown in FIG. 1.

The information display computing device 150 may take various forms. Inone embodiment, the information display computing device 150 may be acomputer workstation having information display modules 151. In otherembodiments, modules 151 may reside on another computing device, such asa web server, and the user directly interacts with a second computingdevice that is connected to the web server via a computer network. Themodules 151 will be described in detail below.

In one embodiment, the information display computing device 150comprises a server, a desktop computer, a workstation, a laptopcomputer, a mobile computer, a smartphone, a tablet computer, a cellphone, a personal digital assistant, a gaming system, a kiosk, an audioplayer, any other device that utilizes a graphical user interface,including office equipment, automobiles, airplane cockpits, householdappliances, automated teller machines, self-service checkouts at stores,information and other kiosks, ticketing kiosks, vending machines,industrial equipment, and/or a television, for example.

The information display computing device 150 may run an off-the-shelfoperating system 154 such as a Windows, Linux, MacOS, Android, or iOS.The information display computing device 150 may also run a morespecialized operating system which may be designed for the specifictasks performed by the computing device 150.

The information display computing device 150 may include one or morecomputing processors 152. The computer processors 152 may includecentral processing units (CPUs), and may further include dedicatedprocessors such as graphics processor chips, or other specializedprocessors. The processors generally are used to execute computerinstructions based on the information display software modules 151 tocause the computing device to perform operations as specified by themodules 151. The modules 151 may include, by way of example, components,such as software components, object-oriented software components, classcomponents and task components, processes, functions, attributes,procedures, subroutines, segments of program code, drivers, firmware,microcode, circuitry, data, databases, data structures, tables, arrays,and variables. For example, modules may include software code written ina programming language, such as, for example, Java, JavaScript,ActionScript, Visual Basic, HTML, C, C++, or C#. While “modules” aregenerally discussed herein with reference to software, any modules mayalternatively be represented in hardware or firmware. Generally, themodules described herein refer to logical modules that may be combinedwith other modules or divided into sub-modules despite their physicalorganization or storage.

The information display computing device 150 may also include memory153. The memory 153 may include volatile data storage such as RAM orSDRAM. The memory 153 may also include more permanent forms of storagesuch as a hard disk drive, a flash disk, flash memory, a solid statedrive, or some other type of non-volatile storage.

The information display computing device 150 may also include or beinterfaced to one or more display devices 155 that provide informationto the users. Display devices 155 may include a video display, such asone or more high-resolution computer monitors, or a display deviceintegrated into or attached to a laptop computer, handheld computer,smartphone, computer tablet device, or medical scanner. In otherembodiments, the display device 155 may include an LCD, OLED, or otherthin screen display surface, a monitor, television, projector, a displayintegrated into wearable glasses, or any other device that visuallydepicts user interfaces and data to viewers.

The information display computing device 150 may also include or beinterfaced to one or more input devices 156 which receive input fromusers, such as a keyboard, trackball, mouse, 3D mouse, drawing tablet,joystick, game controller, touch screen (e.g., capacitive or resistivetouch screen), touchpad, accelerometer, video camera and/or microphone.

The information display computing device 150 may also include one ormore interfaces 157 which allow information exchange between informationdisplay computing device 150 and other computers and input/outputdevices using systems such as Ethernet, Wi-Fi, Bluetooth, as well asother wired and wireless data communications techniques.

The modules of the information display computing device 150 may beconnected using a standard based bus system. In different embodiments,the standard based bus system could be Peripheral Component Interconnect(“PCI”), PCI Express, Accelerated Graphics Port (“AGP”), Micro channel,Small Computer System Interface (“SCSI”), Industrial StandardArchitecture (“ISA”) and Extended ISA (“EISA”) architectures, forexample. In addition, the functionality provided for in the componentsand modules of information display computing device 150 may be combinedinto fewer components and modules or further separated into additionalcomponents and modules.

The information display computing device 150 may communicate and/orinterface with other systems and/or devices. In one or more embodiments,the computing device 150 may be connected to a computer network 190. Thecomputer network 190 may take various forms. It may include a wirednetwork or a wireless network, or it may be some combination of both.The computer network 190 may be a single computer network, or it may bea combination or collection of different networks and network protocols.For example, the computer network 190 may include one or more local areanetworks (LAN), wide area networks (WAN), personal area networks (PAN),cellular or data networks, and/or the Internet.

Various devices and subsystems may be connected to the network 190. Forexample, one or more medical scanners may be connected, such as MRIscanners 120. The MRI scanner 120 may be used to acquire MRI images frompatients, and may share the acquired images with other devices on thenetwork 190. The network 190 may also include one or more CT scanners122. The CT scanners 122 may also be used to acquire images and, likethe MRI scanner 120, may then store those images and/or share thoseimages with other devices via the network 190. Any other scanner ordevice capable of inputting or generating information that can bedisplayed as images or text could be included, including ultrasound,angiography, nuclear medicine, radiography, endoscopy, pathology,dermatology, etc.

Also connected to the network 190 may be a Picture Archiving andCommunications System (PACS) 136 and PACS workstation 138.

Also connected to the network 190 may be an interaction behavior modeldata structure 160 used to store interaction behavior models. In variousembodiments, the interaction behavior model data structure 160 mayreside within PACS System 136, reside within a server accessible on aLAN that is local to the information display computing device 150,and/or reside within a server that is located remote to the informationdisplay computing device 150 and accessible via the Internet. In otherembodiments, interaction behavior model data structure 160 may residelocally, within information display computing device 150. Interactionbehavior model information may be stored in any computer readable formatsuch as a database, flat file, table, or XML file, and may be stored onany computer readable medium, such as volatile or non-volatile memory,compact disc, digital video disc, flash drive, or any other tangiblemedium.

The PACS System 136 may be used for the storage, retrieval, distributionand presentation of images (such as those created and/or generated bythe MRI scanner 120 and CT Scanner 122). The medical images may bestored in an independent format, an open source format, or some otherproprietary format. One format for image storage in the PACS system isthe Digital Imaging and Communications in Medicine (DICOM) format. Thestored images may be transmitted digitally via the PACS system, oftenreducing or eliminating the need for manually creating, filing, ortransporting film.

The network 190 may also be connected to a Radiology Information System(RIS) 140. The radiology information system 140 may be a computerizeddata storage system that is used by radiology departments to store,manipulate and distribute patient radiological information.

Also attached to the network 190 may be an Electronic Medical Record(EMR) system 142. The EMR system 142 may be configured to store and makeaccessible to a plurality of medical practitioners computerized medicalrecords. Also attached to the network 190 may be a laboratoryinformation system 144. Laboratory information system 144 may be asoftware system which stores information created or generated byclinical laboratories. Also attached to the network 190 may be a digitalpathology system 146 used to digitally manage and store informationrelated to medical pathology.

Also attached to the network 190 may be a computer aided diagnosissystem (CAD) 148 used to analyze images. In one embodiment, the CAD 148functionality may reside in a computing device separate from theinformation display computing device 150 while in another embodiment theCAD 148 functionality may reside within the information displaycomputing device 150.

Also attached to the network 190 may be a 3D Processing System 149 usedto perform computations on imaging information to create new views ofthe information, e.g., 3D volumetric display, Multiplanar Reconstruction(MPR) and Maximum Intensity Projection reconstruction (MIP). In oneembodiment, the 3D processing functionality may reside in a computingdevice separate from the information display computing device 150 whilein another embodiment the 3D processing functionality may reside withinthe information display computing device 150

In other embodiments, other computing devices that store, provide,acquire, and/or otherwise manipulate medical data may also be coupled tothe network 190 and may be in communication with one or more of thedevices illustrated in FIG. 1, such as with the information displaycomputing device 150.

As will be discussed in detail below, the information display computingdevice 150 may be configured to interface with various networkedcomputing devices in order to provide efficient and useful review ofmedical examination data that is stored among the various systemspresent in the network. In other embodiments, information displaycomputing device 150 may be used to display non-medical information.

Depending on the embodiment, the other devices illustrated in FIG. 1 mayinclude some or all of the same components discussed above withreference to the information display computing device 150.

Input Devices

There are embodiments of the interaction behavior model in which theuser controls presentation of visual information by interacting with aninput device interfaced to an information display computing device.While embodiments will be described using input devices in which theuser physically interacts with the input device, the systems and methodsdescribed herein can be applied to any input method including:

-   -   Devices that rely on direct physical interaction, including        devices or portions of devices that the user rotates, tilts,        presses, lifts up or down, squeezes, translates and/or touches,        either with a body part or device such as a stylus.    -   Input methods that rely on measurement of a user's muscular or        neural electrical activity.    -   Input methods that sense a user's position, configuration or        motion, including body parts such as extremities, eyes, and        face, for example using a video camera.    -   Input methods that rely on detection of sound, including        recognition of voice commands.

FIG. 2 illustrates an input device 200 that utilizes input related totilt. In this embodiment, the device illustrated has a base 221 and aknob 222 that is manipulated by the user. Knob 222 could instead beattached to something other than a base, such as a large input device,panel or dashboard. In this example, the user may tilt the knob to theleft or right, illustrated by arrows 225 and 226, but in otherembodiments the input device could accept user input in other ways, forexample with translation, rotation, etc., as discussed below.

FIG. 3 is a graph illustrating image display rate verses tile angle ofknob 222 with respect to base 221 (FIG. 2). In this embodiment, theinformation display computing device is configured to display a seriesof images at a rate that is controlled by the tilt that the user appliesto the input device 200.

FIG. 2 and FIG. 3 illustrate changes in the display rate of images asthe user tilts the knob 222 to the right. Although not illustrated, thesame or different behavior might occur when the user tilts the knob 222to the left. For example, tilting the knob 222 to the right of midline230 might increment the number of the images displayed per second whiletilting the knob to the left of midline might decrease the number of theimages displayed per second, where the degree of tilt affects the rateof display of images. Thus, in this embodiment the degree of tilt off ofmidline determines the rate of display of images, while the direction oftilt determines a direction of movement within a series of images, suchas whether the currently displayed image number should be increased(e.g., to move to a next image in an image series) or decreased (e.g.,to move to a prior image in the image series). In other embodiments,other input device controls can be used in similar manners to determinethe display rate of images and the direction of movement between images.

As illustrated in FIG. 2, the speed of image presentation would bedetermined by the degree of tilt from the midline 230 in FIG. 2. Asillustrated in FIG. 2, a tilt between position 230 and 231 would resultin no change in the image displayed. A tilt between position 231 and 232would result in serial presentation of images at a rate of 4images/second, between 232 and 233 a rate of 10 images/second, andbeyond 233 a rate of 20 images/second.

The graph of FIG. 3 includes two different mappings between tilt angleand the action of the information display computing device. Inparticular, line 241 shows the mapping between tilt angle, shown on thehorizontal axis, to image display rate, shown on the vertical axis. Line242 shows another mapping in which the display rate is slower and thetransition between display rates occurs at tilt angles that aredifferent than those utilized within the mapping shown in line 241. Invarious embodiments, different mappings within an interaction behaviormodel might be chosen manually, for example as a user preference, orautomatically, for example based on image size, image content, userfatigue, or other factors as discussed herein.

FIG. 4 illustrates four graphs that each illustrate exemplary mappingsof image display rate as a function of tilt angle for a device such asthe one shown in FIG. 2. In each of the mappings, illustrated by graphs350, 360, 370, and 380, there are two lines illustrating differentinteraction behaviors that may be present in an interaction behaviormodel, but any number of behaviors may be present.

FIG. 5 a shows another example of an embodiment of an interactionbehavior model. In this case the information is represented as textualinformation rather than graphically as in FIG. 3 and FIG. 4. In thisembodiment the interaction behavior model includes three components,behavior settings 390, modifier settings 392 and conditions 394. In thisembodiment, the interaction behavior model defines two differentbehaviors B1 and B2 in the behavior settings 390. In this example, thebehaviors B1 and B2 define rates of displaying images in response to atilt angle of an input device. In other embodiments, behaviors maydefine any other display characteristic and/or feature in response toother defined inputs from one or more input device. In this example, themodifier settings 392 include four modifiers M1-M4 that may be appliedto one of the behaviors B1 or B2 in response to meeting of certain ofthe conditions C1-C6 in the conditions settings 394. Thus, in oneembodiment the conditions are rules that are applied to determine whichof the behaviors should be applied and when the selected behavior shouldbe modified by one of the modifiers. For example, condition C1 indicatesthat B1 is the default behavior. Thus, the display speed defined by B1is used as a default when this interaction behavior model is selected.However, condition C2 indicates that if the exam from which images aredisplayed is a CTA exam, behavior B2 is to be used to define the displayspeed of images. Thus, the display speed may be changed based on thetype of images that are displayed. Condition C3 indicates that amodifier (modifier M1) of the currently selected behavior (e.g., B1 bydefault or B2 if the exam is a CTA) to decrease the display rate by 20%is applied in response to display of an image that is greater than 1megapixel. Accordingly, the behaviors may be modified in response tocharacteristics of the image being displayed. The remaining conditionsin FIG. 5 a illustrate other example criteria by which differentbehaviors may be selected and modifiers to the selected behaviorsapplied. In other embodiments, fewer, more or different components maybe used.

FIG. 5 b illustrates example conditions that may be used in aninteraction behavior model that limits drive speed of a user operatedvehicle. As noted above, interaction behavior models may be used inother contexts beyond image display. FIG. 5 b illustrates one suchexample, but countless other uses of such interaction behavior modelsare within the scope of this disclosure.

In the example of FIG. 5 b, conditions C1-C4 each cap the maximum speedof the vehicle to respective caps based on different criteria. Inparticular, condition C1 caps the speed at 10 mph if the GPS determinesthat the vehicle is at a particular location (e.g., a strip mine in theexample of FIG. 5 b), condition C2 caps the speed at 8 mph if the driverhas less than 2 years experience or has worked more than 8 hours,condition C3 caps the speed at 5 mph if sensors determine that it isdark outside, and condition C4 caps the speed at 5 mph if the mineconditions are designated as dangerous, which may be determined based ona data structure that is updated to include such information in responseto user and/or automated sensor input. Thus, the conditions of FIG. 5 bare based on vehicle speed, user conditions and experience,environmental conditions, and current conditions defined by a thirdparty. In other embodiments, any other criteria may be used to setmaximum vehicle speed (or any other characteristic).

FIG. 5 c illustrates sample data 650 and graph 652 that correlates inputfrom an input device into various outputs that could be interpreted by acomputing device. In this example, three associations between input andoutputs are indicated as behavior 1, behavior 2, and behavior 3,illustrated in tabular form in data 650 and in graph 652. As describedin other embodiments, the choice of which of the output behaviors isused could be based on one or more factors as described herein. Inaddition, these behaviors could be modified based on one or more factorsas described herein.

FIG. 6 is a flowchart illustrating one embodiment of a method ofimplementing an interaction behavior model. In one embodiment, themethod of FIG. 6 is performed by the information display computingdevice 150 of FIG. 1. For ease of explanation, the method of FIG. 6 isdescribed herein with reference to the information display computingdevice 150, with the understanding that in other embodiments the methodmay be performed by any other suitable computing device. Depending onthe embodiment, the flowchart of FIG. 6 may include additional or fewerblocks and/or the blocks may be performed in a different order than isillustrated.

At Initialization block 705, information relevant to the use of theinteraction behavior model is accessed and/or retrieved, such asinformation regarding the user and his preferred interaction behaviormodel. For example, a particular user might prefer the interactionbehavior model shown in FIG. 5 a, while a different user might prefer adifferent interaction behavior model or a similar one with differentdefault setting.

One or more interaction behavior models may be stored in interactionbehavior model data structure 160 shown in FIG. 1 and/or stored, forexample within information display computing device 150. Userpreferences could also be stored in interaction behavior model datastructure 160 or in another data structure, for example within PACSSystem 136 of FIG. 1.

In other embodiments, there are no user specific preferences, butdefault interaction behavior models that could be determined by themanufacturer of the information display computing device or, forexample, set by a site to apply to all users or to groups of users. Forexample, behavior B1 in behavior settings 390 (FIG. 5 a) might beapplied as the default behavior to radiologists in training at a sitewhile behavior B2 might be applied by default to staff radiologists.

In block 710, with a behavior selected (e.g., based on conditions, suchas conditions 394 of FIG. 5 a), the computing system determines if anymodifiers should be applied to the selected behavior and/or if adifferent behavior should be selected. In the example of FIG. 5 a, thisis done via the conditions C2-C6, which indicate application ofmodifiers and/or changes to the current behavior in response to theindicated conditions. Using the conditions of FIG. 5 a, for example, ifthe patient has cancer or is suspected of having cancer, condition C6(section 394 in FIG. 5 a) would result in the application of modifier M2which would slow the image display rate by 50%. Behaviors may beselected and/or customized based on any one or more of the factorslisted above and/or any other factors, and may be applied using acondition/rule system such as in FIG. 5 a, or in other manners.

While some factors such as clinical information may not change duringthe course of viewing the imaging exam, other factors may changedynamically, such as:

-   -   The size of the image being displayed (relevant to condition C3        in the example illustrated in FIG. 5 a).    -   Whether a nearby image has been marked as positive by a Computer        Aided Detection (CAD) system (relevant to condition C4 in the        example illustrated in FIG. 5 a).    -   Image display parameters (relevant to condition C5 in the        example illustrated in FIG. 5 a).

Thus, in some embodiments the behavior and/or modifiers to behaviors areadjusted in real time as a user views medical images. For example, if auser initially views images that are less than 1 megapixel, condition C3is not triggered, but if the user moves to an image in an image seriesor otherwise displays an image that is 1.1 megapixel, condition C3 maybe immediately triggered such that the image is displayed with modifierM1 applied. Information regarding any other factors may be accessedand/or retrieved in block 710, such as the factors listed above and/orany other factors.

In block 715 user input is received via any available input devicesand/or user interactions. For example, in the embodiment of FIG. 2, userinput would include the tilt angle the user has applied to knob 222 andmay include other input, for example from a keyboard, mouse, or otherbuttons on the input device with the tilt knob illustrated in FIG. 2. Asnoted above, the input device of FIG. 2 is provided only as an exampleof an input device—any other input device, even an input device thatdoesn't require direct user contact, such as a motion detection cameraor microphone, may be used in place of the tilt knob.

In block 730, the information display computing device 150 applies theselected interaction behavior model, including any modifications, inorder to determine the appropriate action. For example, in theembodiment of FIG. 5 a this may be accomplished by applying theconditions sequentially to determine the behavior and any modifiers thatwould be mapped to the received user input. For example, consider an MRIof the Brain in a patient with a clinical history of Lung Cancer whenthe user is applying a tilt angle of 11 degrees to the knob of the inputdevice illustrated in FIG. 2. Stepping through the conditionssequentially in the example illustrated in FIG. 5 a:

-   -   Condition C1 is always applied initially so behavior B1 is        initially selected.    -   Condition C2 is false as the exam is an MRI not a CTA, so        behavior B2 is still selected.    -   Condition C3 is false as the MRI images are <1 megapixel/image.    -   Condition C4 is false as CAD was not utilized.    -   Condition C5 is false as the modality is not CT.    -   Condition C6 is true as the history is cancer so Modifier M2 is        applied to the current selected behavior B1.

Having stepped through the conditions in this example, the informationdisplay computing device 150 would determine that behavior B1 withmodifier M2 are to be applied to the received user input. Applying atilt angle of 11 degrees to behavior B1 maps to a display rate of 10images/sec. Applying modifier M2 (activated by condition C6) decreasesthe display rate by 50%, resulting in a display rate of 5 images/second.

In block 740 the determined action is performed. In this example, theinformation display computing device 150 also considers the direction ofthe tilt of knob 226, left vs. right, to determine whether the image tobe displayed within the series is to be incremented or decremented.Assuming the knob is tilted to the right, the image number within theseries is incremented and the resulting image is displayed for 200milliseconds, the determined rate of 5 images/second before a new imageis displayed. The logic loops back to block 710 to repeat the process.

In other embodiments the user has the option of overriding theinteraction behavior model, for example by pressing a key that returnsto the default behavior.

Image Navigation Using Other Input Devices

With the device shown in FIG. 2, the input device senses the degree oftilt of a knob manipulated by the user. Tilt input could also be sensedin other ways, for example by devices that utilize accelerometers,including handheld controllers such as the Wiimote made by Nintendo, andcomputing devices with internal sensors such as the iPhone and iPad madeby Apple. However, other technology, including cameras and othersensors, may be used to sense the physical positions of objects or bodyparts. The embodiments described herein may be used with any devicesand/or methods that are capable of sensing the desired user input, suchas tilt in the embodiment of FIG. 2.

FIG. 7 illustrates a device 310 that includes a display screen and asensor that detects the device orientation. By tilting the device, theuser can provide tilt information to the device in a way that isanalogous to tilting the knob in the input device illustrated in FIG. 2.

In view 320 of FIG. 7, device 310 is shown in a particular position,theta degrees from horizontal. In one embodiment, the user may indicateto the computing device that the current tilt of device 310 is to beassigned as a neutral position, for example by pressing a button on thedevice or touching a control on the touch screen. When in this neutralposition, the computing device would not change images, which isanalogous to a midline position of the knob in FIG. 2. Tilting device320 to angles different than theta, such as is shown in views 322 and324, would be similar to tilting the knob of the device shown in FIG. 2.For example, tilting the device 310 into positions 322 or 324 may beequivalent to tilting the knob of the device shown in FIG. 2 to the leftor the right, respectively, causing an increase or decrease in thenumber of the image displayed within the series, where the degree oftilt from the neutral position may determine the rate of incrementalimage change, as described previously.

In another embodiment, the actual tilt may be mapped to different imagesin the series. For example, in a series with 90 images, positioning thedevice horizontally, such as shown in view 324, may cause display ofimage 1, and positioning the device vertically might cause display ofimage 90, with intermediate angles causing display of intermediateimages in a linear fashion.

In other embodiments the interaction behavior model may be used withleft and right tilting of the device 310. In other embodiments, otherinput devices that are manipulated in other ways, for example asillustrated in FIG. 8, may be used in conjunction with interactionbehavior models.

FIG. 8 illustrates another example input device, in particular, a knob401. In view 410 of FIG. 8, knob 401 of an input device may be tiltedleft-right and/or forward-backward, as illustrated by the curved arrowsabove the knob 401. The systems and methods described herein may beapplied to one or both of these two axes of tilt.

In view 420, a different input mode is shown, where the knob 401 may betwisted clockwise and counterclockwise, where the twisting motion may beindependent from the motions shown in view 410. The systems and methodsdescribed herein may be applied to rotation, where angle or rotation istreated in a way that is similar to angle of tilt, as previouslydescribed.

In view 430, the knob 401 may be translated by the user forward-backwardand/or left-right. The systems and methods described here may be appliedto these directions as well.

In view 440, the knob 401 may be translated by the usersuperiorly/inferiorly, perpendicular to the direction of translationshown in view 430. This systems and methods described herein may beapplied to this input mode as well.

In the device's shown in FIG. 8, as the user translates, tilts, and/orrotates the knob of the input device from a neutral position to changethe angle of tilt, angle of rotation, and/or degree of translation, thedirection, timing, and/or other factors of image viewing (or othercomputer functionality) may be adjusted according to an interactionbehavior model. Depending on the embodiment, the input device mayinclude one or a combination of multiple of the movement capabilitiesdiscussed with reference to knob 401.

FIG. 9 illustrates movements of a mouse 501 that can be used to controla user interface for viewing images. In one embodiment, the user holdsdown a button 502 on the mouse 501 and moves the mouse 501 forward(motion indicated by arrow 505) or backward (motion indicated by arrow506) on a surface to cause images to change. Typically a fixed incrementin physical mouse position results in a fixed change in the number ofthe image displayed. For example, forward motion of 2 mm might result inan increase in image number and backward motion of 2 mm might result ina decrement in image number. Movement of a mouse wheel 503 may providesimilar inputs that are usable to provide image navigation commands.

Another system that may make use of interaction behavior models might bea touch screen that is separate or integrated with a display device. Forexample, FIG. 10 illustrates a handheld computing device 521 and FIG. 11illustrates a larger tablet computer 601 with an integrated display andtouchpad. With reference to FIG. 10, a stylus 522 may be moved in thedirections illustrated by arrows 525 and 526 in order to navigatebetween images. Similarly, a finger could be utilized instead of astylus, for example in the case of a smartphone or the tablet computer601.

As illustrated in FIGS. 9, 10 and 11, a user can move the mouse 501,roll the mouse wheel 503, move a trackball, or move a stylus or fingeron the touchpad of device 521 or device 601 in order to provide userinput that could be mapped by an interaction behavior model, for exampleto navigate through images, where an incremental change in the degree ofrotation of a trackball or incremental movement of the mouse position ora finger or stylus on a touch screen causes the information displaycomputing device to change the image displayed (and/or other displaycharacteristics, such as speed of changing between images).

FIG. 12 illustrates four graphs 450, 460, 470, 480 that illustrateinputs and outputs of an interaction behavior model over time. Inparticular, FIG. 12 illustrates mouse position in graph 450, mouse speedin graph 460, image number in graph 470, and image change rate in graph480, each with reference to a common time period and user interactionwith the mouse. Graph 450 shows an example of mouse position over time,for example along the direction of movement indicated by arrow 505 ofFIG. 9. In this example, the user moves the mouse forward first at aslow rate (segment 451), then a medium rate (segment 452) and finally ata fast rate (segment 453). The rate of movement for these three timeperiods is shown in view 460, as mouse speeds 461, 462, and 463.

The line segments 471, 472, and 473 in the graph shown in graph 470indicate the number of the image that would be displayed from a seriesof images as a function of time as a result of the mouse movementgraphed in view 450, where a fixed incremental movement of mouseposition results in a fixed incremental change in image number. The rateof change of image number that corresponds to the image number graph 470shown in the graph 480, where image change rate 481 corresponds tosegment 471, image change rate 482 corresponds to segment 472, and imagechange rate 483 corresponds to segments 473.

In one embodiment, an interaction behavior model may be applied to capthe maximum rate of image change, illustrated in the graph 480 by the“m” on the vertical axis. For example, a behavior and/or a modifier maybe defined that sets a maximum image change rate to m. Therefore, theimage change rate 483 would be reduced to the image change rate 484(which is equivalent to m). This would result in a change in the imagenumber over time illustrated in the graph of view 470, where the thirdsegment of the graph would be segment 474 rather than segment 473.

While the embodiment of FIG. 12 was described using incremental movementof a mouse, in other embodiments other input devices could be usedincluding input on a tablet by finger or stylus, rotation of atrackball, rotation of a mouse wheel or knob, and other input devicesthat sense change in position or orientation.

In one embodiment, an input device, such as a joystick, may be used toprovide commands. A joystick, or other device wherein movement returnsto an original home position when the user releases the device, may beused in a similar manner as discussed above with interaction behaviormodels. Such devices may detect one or more of tilt, translation, and/orrotation of the joystick.

Example Digital Pathology

FIG. 14 illustrates computer monitor 680 displaying a pathology image,for example from a Pathology Information System. Such a pathology imagemay be much larger than can be displayed at full resolution on acomputer monitor. For example, a digital pathology image as might beacquired using a digital slide scanner, represented by image 690, mightbe 100,000×100,000 pixels in size. However, the display resolution of acomputer monitor, represented by monitor 680, might be on the order of1,000×1,000 pixels.

If pathology image 690 is to be displayed at full resolution on monitor690, only a fraction of image 690 may be displayed at any one time.Rectangle 692 represents that portion of image 690 displayed on monitor680, a viewport into the image. The portion being viewed may be changedinteractively by the user, represented by the black arrows along thesides of viewport 692.

In one embodiment, the user utilizes a knob 685 (or any other inputdevice), shown with four black arrows indicating that the user cantranslate the knob. When the computing device senses translation ofinput device 685, that input is used to translate the position ofviewport 692 into image 690, allowing the user to interactively displaydifferent portions of image 690.

Just as it is possible for users to display images at rates that exceedtheir perceptual ability to accurately interpret them, it would bepossible for a user to translate the monitor viewport 692 at rates thatexceed his ability to accurately interpret the information beingdisplayed on monitor 680. Therefore, the systems and methods describedherein may be used to map the input from the user input device 685 intothe speed of translation of viewport 692.

In other embodiments, other input device modes, such as the up-downtranslation shown in view 440 of FIG. 8, may be used to change themagnification of the image displayed on monitor 680. This allows pan andzoom of an image to be controlled by a single input device, with up-downtranslation controlling zoom and left-right/forward-backward translationcontrolling pan.

There are many fields where image size exceeds the display resolution ofthe computing device and in other embodiments, other types of images maybe used, such as satellite imagery, telescope imagery, seismic imagery,and mammography.

Example 3D and Image Rendering

In medical imaging and other areas imaging data may be processed toproduce 3D or other rendered images. In medical imaging, imaginginformation may be processed to create 2D images that are at planesother than the plane in which the images were acquired. Volumetricallyacquired imaging information, for example with CT, MRI and ultrasound,may be processed to create 2D or 3D images, including 3D volumerendering, surface rendering, multiplanar reformatted images (MPR), andmaximum intensity projection (MIP) images. In some cases, 3D volumerendered images may be used to visualize the internal surfaces ofstructures, such as endoluminal imaging of the GI tract with CTcolonography as well as endoluminal imaging of airways, vessels andother structures.

Embodiments described herein may be used with these and/or other typesof images. For example, the speed with which a user travels through a 3Dvolume rendered structure, such as the inside of the colon with CTcolonography, may be controlled with the illustrative systems andmethods described herein. For example, the speed with which a usertraverses images of a colon may be modified as a function of the user'sexpertise, the roughness of the internal surface of the colon, and/orthe presence of regions marked as suspicious or abnormal by a computeraided diagnosis system (CAD) as discussed below.

Example Computer Aided Diagnosis (CAD)

Imaging information may be analyzed by computers to detect regions ofinterest. In medical imaging this is known as Computer Aided Diagnosis(CAD). Examples include the detection of cancers in mammograms,detection of lung nodules in chest CT, detection of polyps in CTcolonography, and detection of abnormal cells in pap smears.

Generally, CAD systems are not sufficiently accurate to make a finaldiagnosis, but rather detect suspicious areas for further scrutiny bythe expert human reader. Marks of some sort may be superimposed on theimages to indicate to the human reader the regions marked as suspiciousby the CAD system. These marks could be placed at the suspiciouslocations in the originally acquired images, for example in mammography,or rendered images, for example 3D or MPR images in the case of CTcolonography.

In one embodiment, an interaction behavior model may be used with CAD tomodify the display of various types of visual information. For example,in the case of a user viewing a series of 2D images such as a chest CT,the rate of image display could be slowed in the regions marked by CAD,whether or not marks are displayed on the images to indicate the regionsmarked by CAD. For example, in a series of images, the maximum imagedisplay rate could be cut by 50% (e.g., and/or required magnificationlevel, contrast level, etc.) for images within 10 mm of an image thathas one or more regions within it marked by CAD. Depending on theembodiment, such changes in display rate might be expressed in modifiersthat are responsive to the indications conditions in an interactionbehavior model. In another example, the maximum image display rate couldbe modified as a result of an interaction behavior model for imageswithin a certain distance of a region marked by CAD, regardless of theimage's 3D spatial orientation.

FIG. 15 a illustrates a series of images 820-829, each representing aplane of imaging through a region of anatomy. The thickness of theanatomy depicted in images could vary, from contiguous slices (no gapbetween the slices), to slices that overlap, to slices having a gapbetween the images. For the purpose of illustration, image 824 is shownwith an associated CAD marker 830. By way of example, images might beacquired at 1 mm intervals and an interaction behavior model might beselected (and or modified by one or more modifiers) to:

-   -   Cap the display rate at 2 images/second for images within 2 mm        of a slice with a CAD mark.    -   Display images with a CAD mark for a minimum of 1 second.

With this example behavior and assuming 1 mm slice spacing, display ofimages 822, 823, 825 and 826 would be capped at a display rate of 2images/second and image 824 would be displayed for a minimum of 1 secondwhen the user displayed those images.

An embodiment of an interaction behavior model may be used in mammotomosynthesis, as illustrated in FIG. 16. FIG. 16 illustrates a seriesof images 850 of different positions of a breast, and a larger view of asubset 855 of those images. Marker 856 is displayed by the computingdevice and indicates a location marked as suspicious by CAD within image857. As discussed above, an embodiment of the interaction behavior modelmay be configured to alter the display of images in proximity to imagesmarked by CAD, in this example image 857. By way of example, images inthe exam that are within a 10 mm distance from an image marked by CAD,image 857 in this example, might be displayed for a minimum of 2seconds. In other embodiments, the minimum display time or maximum imagedisplay rate might be applied only to these images when they aredisplayed for the first time to the user. In other embodiments, othercharacteristics of image display might be changed for images inproximity to images with CAD markers, e.g., magnification, colormapping, brightness, and/or contrast. Other embodiments can also beapplied to other types of imaging where CAD may be utilized, e.g.,breast MRI, chest CT, and/or CT colonography.

In the case of reformatted images, for example thick-slice MIP appliedto chest CT, an embodiment may be used to control the rate of movementof the plane of reconstruction through the imaging volume. For example,the rate of movement of the plane in mm/sec could be capped at 5mm/second when the reconstruction plane is within 20 mm of a region inthe imaging volume marked by CAD.

FIG. 15 b shows aspects of embodiments of an interaction behavior modelin which a region of an imaging volume has been marked by CAD. Forexample, a CAD marker 831 is shown in a spatial position within thesuperior aspect of the imaging volume 840. A slab of the imaging volumebeing reconstructed by thick-slice MIP or another technique is shown asshaded volume 841. For the example above, the speed that the user couldmove the location of reconstruction slab 841 would be capped at 5mm/second when the reconstruction slab was within 20 mm of the regionmarked by CAD, in this example region 831.

In the case of 3D volume rendering, for example, endoluminal imagingsuch as CT colonography, an embodiment may be used to control visualdisplay and 3D rendering. For example, CAD may be used to marksuspicious regions on the inner surface of the colon and it is importantthat the user be aware of these regions. In one embodiment, the ratethat the user traverses the colon via endoluminal 3D volume renderingmay be automatically controlled by the presence of CAD markers orlocations marked as suspicious by CAD, whether or not markers aredisplayed, for example slowing the rate of “movement” when the regionbeing viewed is in proximity to a CAD mark. It is possible for an areaof interest marked by CAD to be hidden from view because of thecomplexity of the inner surface of the colon, for example requiring theuser to look sideways or backwards. In one example, the user is notpermitted to move further than 2 cm beyond the CAD mark until thatregion marked has been displayed on the computer device. This mayrequire the user to change his view within the colon so that he “looks”sideways or backwards.

In another embodiment, interaction behavior related to image translationmight be controlled by the presence of CAD markers. In the exampleembodiments described above with reference to FIG. 14, an image 690could have a CAD marker 693. The rate of translation of viewport 692could be automatically slowed when it is in the proximity of CAD marker693. In other embodiments, a reader could not terminate viewing of image690 until all CAD markers were viewed.

Example Control of Machines

In other embodiments, the interaction behavior model may be used tocontrol how user input via an input device is mapped into physicalactions controlled by a computing device such as a machine or vehicle,rather than display of information. For example, in various embodimentsthe interaction behavior model could be used to control how user inputvia a foot pedal or other input device controls a car, boat, aircraft,spacecraft, submarine, robot or drone.

For example, FIG. 5 b is an embodiment of an interaction behavior modelthat could be used to control the behavior of a strip mining truck'sspeed as a function of user input and several conditions. In thisexample, the maximum speed of the truck is modified based on a number offactors including the physical location of the truck, the experience ofthe driver, the potential for driver fatigue based on number of hoursworked that day, and environmental conditions.

Summary

All of the methods and processes described above may be embodied in, andpartially or fully automated via, software code modules executed by oneor more general purpose computers. For example, the methods describedherein may be performed by an information display computing deviceand/or any other suitable computing device. The methods may be executedon the computing devices in response to execution of softwareinstructions or other executable code read from a tangible computerreadable medium. A tangible computer readable medium is a data storagedevice that can store data that is readable by a computer system.Examples of computer readable mediums include read-only memory,random-access memory, other volatile or non-volatile memory devices,CD-ROMs, magnetic tape, flash drives, and optical data storage devices.

While the methods described herein are typically implemented as softwaremodules, they may alternatively be embodied in specialized computerhardware or firmware. For example, certain methods, portions of methods,and/or combinations of methods may be implemented in hardware modulescomprising programmable units, such as programmable gate arrays (e.g.,FPGAs) or application specific integrated circuits (e.g., ASICs). Theresults of the disclosed methods may be stored in any tangible computerreadable medium.

What is claimed is:
 1. A computing system comprising: one or morehardware processors configured to execute software instructions storedin modules; and a tangible computer readable medium storing modulesconfigured for execution by the one or more hardware processors, themodules configured to: access a data structure storing conditionsassociated with respective behavior models, each of the behavior modelsindicating associations between two or more rates of display of medicalimages and a corresponding two or more levels of interaction, each ofthe two or more levels of interaction indicated by movements of an inputdevice of the computing system; select a behavior model based on one ormore characteristics of: the computing system, an environmentsurrounding the computing system, the display device, bandwidthavailable to the computing system, the medical images, a patient, amedical history of the patient, and/or an input device; and in responseto a first movement of the input device indicating a first level ofinteraction: determine a first rate of display of medical images basedon an association between the first level of interaction and one of thetwo or more rates of display of medical images in the selected behaviormodel.
 2. The computing system of claim 1, wherein the modules arefurther configured to: identify one or more modifications that arematched by one or more characteristics, wherein one or more of theidentified modifications indicate an adjustment to the determined firstrate of display of medical images, wherein the determined first rate atwhich the medical images are displayed is modified by the adjustment. 3.The computing system of claim 2, wherein the adjustment indicates apercentage increase or decrease or an absolute time period.
 4. Thecomputing system of claim 2, wherein one or more of the identifiedmodifications include a maximum rate at which the medical images aredisplayed.
 5. The computing system of claim 1, wherein the level ofinteraction is determined based on one or more of an angle of a knob ofthe input device with reference to a neutral position; a pressureapplied, a speed of movement, a position, an incremental movement, atilt angle, a translation, and/or a rotation of the input device or aportion of the input device; a voice command received by the inputdevice; a volume level associated with the input device; a tone receivedby the input device; and/or a frequency of oral commands received by amicrophone of the input device.
 6. The computing system of claim 1,wherein the first level of interaction indicated by the first movementof the input device includes a direction of interaction with the inputdevice, wherein the medial images are displayed at the determined ratein a direction that is determined based on the direction of interactionwith the input device.
 7. The computing system of claim 6, wherein thedirection of interaction is forward in response to a first direction ofinteraction with the input device and backward in response to a seconddirection of interaction with the input device.
 8. The computing systemof claim 1, wherein the modules are further configured to: in responseto a second movement of the input device indicating a second level ofinteraction: determine a second rate of display of medical images basedon an association between the second level of interaction and one of thetwo or more rates of display of medical images in the selected behaviormodel.
 9. A non-transitory computer readable medium having softwaremodules stored thereon, the software modules configured for execution bya computing system having one or more hardware processors in order to:access a data structure storing conditions associated with respectivebehavior models, each of the behavior models indicating associationsbetween two or more rates of display of medical images and acorresponding two or more levels of interaction, each of the two or morelevels of interaction indicated by movements of an input device of thecomputing system; select a behavior model based on one or morecharacteristics of: the computing system, an environment surrounding thecomputing system, the display device, bandwidth available to thecomputing system, the medical images, a patient, a medical history ofthe patient, and/or an input device; and in response to a movement ofthe input device indicating a level of interaction: determine a rate ofdisplay of medical images based on an association between the level ofinteraction and one of the two or more rates of display of medicalimages in the selected behavior model.
 10. The non-transitory computerreadable medium of claim 9, wherein the software modules are furtherconfigured for execution in order to: identify one or more modificationsthat are matched by one or more characteristics, wherein one or more ofthe identified modifications indicate an adjustment to the determinedrate of display of medical images, wherein the determined rate at whichthe medical images are displayed is modified by the adjustment.
 11. Thenon-transitory computer readable medium of claim 10, wherein theadjustment indicates a percentage increase or decrease or an absolutetime period.
 12. The non-transitory computer readable medium of claim10, wherein one or more of the identified modifications include amaximum rate at which the medical images are displayed.
 13. Thenon-transitory computer readable medium of claim 9, wherein the level ofinteraction is determined based on one or more of an angle of a knob ofthe input device with reference to a neutral position; a pressureapplied, a speed of movement, a position, an incremental movement, atilt angle, a translation, and/or a rotation of the input device or aportion of the input device; a voice command received by the inputdevice; a volume level associated with the input device; a tone receivedby the input device; and/or a frequency of oral commands received by amicrophone of the input device.
 14. The non-transitory computer readablemedium of claim 9, wherein the level of interaction indicated by thefirst movement of the input device includes a direction of interactionwith the input device, wherein the medial images are displayed at thedetermined rate in a direction that is determined based on the directionof interaction with the input device.
 15. The non-transitory computerreadable medium of claim 14, wherein the direction of interaction isforward in response to a first direction of interaction with the inputdevice and backward in response to a second direction of interactionwith the input device.
 16. A method comprising: accessing, by acomputing system having one or more hardware processors, a datastructure storing conditions associated with respective behavior models,each of the behavior models indicating associations between two or morerates of display of medical images and a corresponding two or morelevels of interaction, each of the two or more levels of interactionindicated by movements of an input device of the computing system;selecting, by the computing system, a behavior model based on one ormore characteristics of: the computing system, an environmentsurrounding the computing system, the display device, bandwidthavailable to the computing system, the medical images, a patient, amedical history of the patient, and/or an input device; and in responseto a movement of the input device indicating a level of interaction:determining, by the computing system, a rate of display of medicalimages based on an association between the level of interaction and oneof the two or more rates of display of medical images.
 17. The method ofclaim 16, further comprising: identifying one or more modifications thatare matched by one or more characteristics, wherein one or more of theidentified modifications indicate an adjustment to the determined rateof display of medical images, wherein the determined rate at which themedical images are displayed is modified by the adjustment.
 18. Themethod of claim 17, wherein one or more of the identified modificationsinclude a maximum rate at which the medical images are displayed. 19.The method of claim 16, wherein the level of interaction is determinedbased on one or more of an angle of a knob of the input device withreference to a neutral position; a pressure applied, a speed ofmovement, a position, an incremental movement, a tilt angle, atranslation, and/or a rotation of the input device or a portion of theinput device; a voice command received by the input device; a volumelevel associated with the input device; a tone received by the inputdevice; and/or a frequency of oral commands received by a microphone ofthe input device.
 20. The method of claim 16, wherein the level ofinteraction indicated by the movement of the input device includes adirection of interaction with the input device, wherein the medialimages are displayed at the determined rate in a direction that isdetermined based on the direction of interaction with the input device.21. The method of claim 20, wherein the direction of interaction isforward in response to a first direction of interaction with the inputdevice and backward in response to a second direction of interactionwith the input device.